Several versions of the dot probe detection task are frequently used to assess maladaptive attentional processes associated with a broad range of psychopathology and health behavior, including eating behavior and weight. However, there are serious concerns about the reliability of the indices derived from the paradigm as measurement of attentional bias toward or away from salient stimuli. The present paper gives an overview of different attentional bias indices used in psychopathology research and scrutinizes three types of indices calculated from a (...) pictorial version of the dot probe task to assess food-related attentional biases in children and youngsters with and without obesity. Correlational analyses reveal that dynamic scores are dependent on general response speed. Reliability estimates are low for the traditional and probability indices. The higher reliability for the dynamic indices is at least partially explained by general response speed. No significant group differences between youth with and without obesity are found, and correlations with weight are also non-significant. Taken together, results cast doubt on the applicability of this specific task for both experimental and individual differences research on food-related attentional biases in youth. However, researchers are encouraged to make and test adaptations to the procedure or computational algorithm in an effort to increase psychometric quality of the task and to report psychometric characteristics of their version of the task for their specific sample. (shrink)